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SUMMARY 


The  Air  Force  enlisted  training  system  is  an  extensive  c^jeration.  The 
thousands  of  people  vho  enter  the  Air  Force  each  year  undergo  sene  type  of 
training  at  various  points  in  their  careers.  This  training  is  provided  in 
different  settings  such  as  resident  technical  training  courses,  Field 
Training  Detachments  (FTDs) ,  on-the-job  training  (OUT) ,  corre^xjndence 
courses,  and  others.  Many  inportant  decisions  vAiich  influence  training 
outccjnes  must  be  made  by  the  various  Air  Force  agencies  re^)onsible  for 
enlisted  personnel  utilization  and  training.  Currently,  coordination  does 
not  tate  place  among  the  various  agencies  responsible  for  determining 
vhat,  vhen,  and  vhere  to  train.  The  Training  Decisions  System  (TDS)  will 
be  a  four-subsystem  oonputerized  training  decisions  aid.  The  Thsk 
Characteristics  Subsystem  (TCS)  will  help  decision-makers  to  determine 
vhere  and  how  long  tasks  should  be  trained,  vhile  the  Field  Utilization 
Subsystem  (FUS)  will  help  determine  patterns  of  personnel  utilization. 

The  Resource/Cost  Subsystem  (PCS)  will  provide  estimates  of  training  cost 
and  capacity  for  personnel  utilization  and  training  alternatives. 

Finally,  integrating  these  issues  to  determine  cptinal  training  solutions 
will  be  performed  by  the  Integration  and  Optimization  Subsystem  (lOS) . 

The  HJS  has  three  major  conponents.  The  Curr^t  Utilization  and 
Training  (U&T)  Pattern  Component  describes  the  currently  existing  U&T 
pattern  of  an  Air  Force  specicilty  (AFS) .  An  Alternative  U&T  Pattern 
Cortponent  provides  logical  possible  changes  to  the  current  pattern. 
Finally,  the  Management  Preferences  Component  edlcws  comparison  of 
preference  data  across  managerial  grotps.  The  task  of  constructing 
current  U&T  patterns  first  reqiiired  identifying  jobs  in  terms  of  the  Task 
Training  Modules  (TIMS)  frorni  the  TCS.  Air  Force  job-typing  procedures 
were  used  for  this  tcisk.  The  effort  to  construct  the  job  flows  involved 
dynamic  Cross-KPAIH  Analysis  to  determine  vhere  re^»ndents  were  during 
two  survey  administrations  (i.e. ,  time  1-time  2) .  Other  procedures  were 
tested,  among  them  the  use  of  Subject-Matter  Experts  (SMEs)  to  draw  U&T 
patterns  and  the  use  of  a  U&T  History  Survey.  The  current  U&T  patterns 
were  the  foundation  for  alternative  U&T  patterns  developed  fremn  functional 
and  training  manager  inputs.  Finally,  the  alternative  U&T  patterns, 
contained  in  survey  booklets,  were  presented  to  iianagers  as  a  means  for 
collecting  managers'  preference  data.  The  procedures  developed  for  the 
FUS  provided  usable  U&T  patterns  for  all  four  AFSs  involved  in  the  stuc^. 
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PREFACE 


The  Field  Utillzaticn  Subsystem  (FUS)  is  one  cxxcponent  of  the  Training 
Decisions  System  (1C6)  research  and  development  (R&D)  effort.  Ihe  TD6  is 
a  nulti-year,  multimillion  dollar  R&D  effort  consisting  of  four  research 
ocnponents  and  is  sponsored  by  HQ  USAF/DPPT  and  HQ  ATC/XPC.  Uiis  effort 
is  being  aooccplished  under  Project  7734  and  is  designed  to  provide  a  more 
integrated  approach  to  training  problems. 

A  project  of  this  magnitude  rec^iires  the  cooperation  and  dedication  of 
many  people.  In  this  respect  we  adkncwledge  the  professional 
ocartributions  made  by  Mr.  Wayne  Archer  of  the  Marpcwer  and  Perscamel 
Division,  Air  Force  Human  Resources  Laboratory.  Special  thanks  go  to  aill 
the  Major  Ckatmands  for  their  assistance  in  maMng  available  the 
Subject-Matter  Ejperts  who  were  instrummital  in  providing  the  data  for  the 
RJS  and  to  the  training  and  functional  managers  of  the  AFSs  involved  in 
this  study. 
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1.0  ItTrRODUCrK^ 


The  Field  Utilization  Subsystem  (HJS)  is  one  of  four  basic  subsystems 
of  the  prototype  Training  Decisions  Syst^  (TDS) ,  vhich  is  being  developed 
under  contract  for  the  Air  Force  Hunan  Resources  Laboratory  (AFHRL)  •  It 
has  three  main  purposes  within  the  TDS: 

1.  It  describes  the  current  Utilization  and  Training  (U&T)  pattern  for 
a  specific  Air  Force  speciality  (AFS) .  U&T  pattern  develcpnent  involves 
identifying  the  present  structure  of  work  and  training  in  a  career  field, 
determining  the  manner  of  personnel  flow  between  jcias,  and  describing 
relationships  between  training  and  the  work  performed. 

2.  It  provides  a  means  for  develcping  alternatives  to  the  current  U&T 
pattern  that  mi^t  inprove  some  aspect  of  work  or  training  in  the  career 
field.  Exairples  include  ideas  that  mi^t  result  in  lowered  training 
costs,  enhanced  readiness,  or  elimination  from  the  career  field  of  those 
tasks  vhich  would  be  more  appropriately  performed  by  members  of  ainother 
career  field. 

3.  It  provides  methods  to  assist  Air  Force  managers  of  various  sorts 
(e.g. ,  functional  managers,  training  managers,  personnel  managers)  in 
rating  the  current  pattern  and  eiltematives  according  to  preference, 
compiles  preference  scores  from  the  ratings,  and  examines  the  reasons  vhy 
some  alternatives  are  preferred  over  others  by  the  different  sorts  of 
managers. 

This  paper  documents  the  research  and  development  (R&D)  on  the 
procedures,  software,  and  instrumentdtion  of  tiie  FUS  to  support  these 
ctojectives. 


1.1  Other  TDS  Subsystems 

In  addition  to  its  intrinsic  purposes  of  describing  and  comparing  U&T 
patterns,  the  FUS  must  also  interface  with  other  IDS  subsystems.  First, 
FUS  jobs  must  be  described  and  manipulated  in  terms  of  Task  Training 
Modules  (TTMs)  produced  by  the  Task  Characteristics  Subsystem  (TCS) .  TTMs 
are  groups  of  tasks  which  can  and  should  be  trained  together.  Perrin, 
Khi<^t,  Mitchell,  Vaui^ian,  and  Yadrick  (1987)  described  the  conc^jt  of 
TTMs,  including  their  justification,  development,  validation,  and  other 
uses  within  the  TDS. 

The  FUS  must  also  present  information  about  jcto  structures  and 
personnel  flow  in  terms  suitable  for  use  by  the  Resource/Ctost  Subsystem 
(RCS) .  Briefly,  the  RCS  (Rueter,  1986)  determines  the  training  resources 
required  to  svpport  the  current  and  suggested  alternative  U&T  patterns, 
and  the  costs  2issociated  with  providing  those  resources.  This  allows  U&T 
patterns  to  be  compeired  on  the  basis  of  training  cost  to  the  Air  Force. 


Finally,  the  information  supplied  by  the  FUS  must  be  fed  to  the 


IntegraticaVOptimization  Subsystem  (lOS) .  The  lOS  is  the  "meeting  point" 
product  must  be  passed  fron  one  subsystem  to  another.  The  106 
ocntains  model ing/optimizaticai  routines  that  will  aLLlow  "what  if" 
projec±ions;  it  can  optimize  vairiables  in  one  subsystem  with  respect  to 
ccxTstraints  inposed  by  variables  in  the  same  or  another  subsystem.  The 
lOS  will  edso  provide  user  interface  software,  for  example,  to  control 
report  selecticai  opticxis. 


1.2  The  TPS  and  Air  FoTxse  Training  Decisicyi^lakina 


Air  Force  enlisted  perscnnel  reqvdre  different  sorts  of  training  at 
many  points  in  their  caireers,  ard  the  Air  Force  operates  a  leirge-scale 
training  establishment  to  me^  their  needs  (Mitchell,  Sturdevant,  Vaughcin, 
&  Pueter,  1986) .  Various  factors  impact  the  naneigement  of  this  training 
system.  For  example,  training  is  provided  in  several  settings,  including 
basic  and  advanced  resident  training  schools.  Field  Training  Detachments 
(FTDs) ,  Career  Development  Courses  (CDCs) ,  on-the-job  training  (COT) , 
Mobile  Training  Teams  (MTTs) ,  and  o^ers.  Moreover,  management  of  these 
settings  is  the  responsibility  of  more  than  one  authority,  creating 
problems  of  coordination.  Various  managers  in  Air  Training  Command  (ATC) 
are  directly  responsible  for  resident  training  and  CDCs,  vhereas  COT  is  a 
responsibility  of  the  Air  Force  Manpower  and  Personnel  Center  (AFMPC) , 
major  oantands,  and  first-line  sipervisors. 


An  importcint  training  management  issue  ccnoems  training  edlocation; 
that  is,  \ft)o  should  be  trained  on  vAiich  job  requirements  in  what  setting. 
Decisions  regarding  this  issue  depend,  in  turn,  upon  assumptions  about  how 
airmen  will  be  used;  i.e.,  the  structure  of  the  jobs  they  will  have  to 
perform.  For  this  reason,  other  managers  besides  those  chiefly 
interested  in  training  must  be  concerned  with  training  issues  and  must  be 
involved  in  making  important  decisions  regarding  training.  Finally,  the 
costs,  resource  requirements,  and  resource  capacities  available  in  the 
various  settings  should  properly  be  wei^ied  vhen  training  decisions  are 
made.  Although  data  of  this  sort  are  available  for  some  training 
settings,  few  objective  data  are  available  ocnoeming  costs  and  capacities 
for  other  settings,  such  as  GOT. 


At  present,  some  ooordinaticn  does  taka  place  eonong  tlie  vaurious 
managers  vhose  decisions  determine  or  influence  the  training  system.  For 
example,  Speci^LLty  Training  Standards  (STSs)  serve  as  contracts  between 
ATC  eUTd  other  Air  Force  commands,  specifying  vhat  training  will  be 
provided  in  formal  resident  courses  and  vhat  will  be  provided  elsevhere. 
Also,  U&T  Workshops  have  become  regulcur  events.  These  cdLlow  managers 
oonoemed  with  irarpcwer  and  persoTnel  utilization  to  meet  those  concerned 
with  training  in  order  to  make  coordinated  decisions.  The  Air  Force's 
Ooctpationed  Surv^  (OS)  program  supports  the  process  by  providing 
managers  with  some  r^evant  data.  In  addition.  Training  Planning  Team 
(TTT)  meetings  taJce  a  long-term  view  of  the  i^ecialty,  especieilly  with 
regard  to  new  weapcai  system  training,  while  RTVET-WDEKPOPCE  meetings 
address  the  ftmcticaally  oriented  initiative  to  consolidate  personnel 
utilization  and  training  (Mitchell  et  eil.,  1986). 


Despite  these  ooordinatj.on  activities,  training  decisions  made  by 
managers  are  often  informal  and  less  than  optimal.  To  a  large  extent. 


this  IS  due  to  the  fact  that  much  of  the  information  required  for  better 
deci^xOTS  remains  unavailable,  much  of  the  available  information  is  not 
suitably  aggregated,  and  data  frcm  various  sources  are  not  easily 
corpared. 

Ihe  TOS  will  serve  as  em  aid  for  inproved  decision-ma)cing  in  the  eurecis 
of  marpcwer  and  personnel  utilization  cind  training,  peu^ly  by  providing 
the  required  data  to  managers.  Sane  of  this  informatics  is  new,  but  much 
old  informatics  has  been  collected  into  a  single  data  base  for  the  first 
time  auid  aggregated  in  new  emd  useful  ways.  The  TDS  atlso  provides 
decisics-makers  with  additional  capabilities,  such  as  models  for 
projec±ing  the  future  effects  of  vcuricus  constraints  and  assunptions,  and 
the  future  consecjuenoes  of  their  decisions.  Maragers  will  be  better  able 
to  cissess  the  broad  range  of  effects  resulting  frcm  vcurious  policy 
decisions  and  to  nake  marpower,  perscsnel,  cind  training  decisicss  vftiich 
best  meet  multiple  policy  objectives. 

In  general,  the  TDS  will  acxonplish  these  cijjectives  by  making 
available  and  expressing  relationships  among  a  nuitber  of  training, 
marpcwer,  and  personnel  utilization  veuriables.  The  following  types  of 
variables  eune  explicitly  represented  within  the  TDS: 

1.  Task  Characteristicxs 

2.  Task  Allocaticxis  to  Training  Settings 

3.  Managers'  Preferences  for  Task  Allccations  to  Training  Settings 

4.  Times  Required  bo  Train  Tasks  in  Various  Setting  Allocations 

5.  Utilization  and  Training  Patterns 

-Jobs  and  Associated  Tasks 

-Training  States  (e.g. ,  FTDs,  CDCs,  ResicJent  Courses) 

-Transiticxi  Probabilities  Between  Jobs  and  Training  States 
-Nuntoers  of  Airmen  in  Training  emd  Jcb  States 

6.  Memagers'  Preferences  for  U&T  Patterns 

7.  Training  Costs 

8.  Training  Resource  Requirements 

9.  Training  Capacities 


The  PUS  has  three  major  cxnponents.  The  Current  U&T  Pattern  Conponent 
prcxvicJes  a  description  of  the  pattern  which  cairrently  exists  in  an  AFS. 

The  Alternative  U&T  Pattern  Carponent  produces  cJescxipticais  of  alternative 
U&T  patterns  which  are  or  may  be  of  interest  to  managers  cxoncxemed  with 
the  AFS.  Finedly,  the  Management  Preferences  Ccnixcaxent  gatiiers  and 
eurranges  data  cxonceming  managers'  preferences  between  the  present  and 
various  alternative  patterns  developed  in  the  first  two  ccrpcxxents. 


In  gaieral,  a  U&T  pattern  xs  a  dynamic  model  reflecting  how  airmen 
move  or  mi^t  move  throu^  jobs  and  training  states  in  a  speci2dty.  Thus, 
the  first  characteristic  of  a  U&T  pattern  is  the  set  of  jobs  in  the 
pattern.  For  present  purposes,  a  job  is  defined  as  a  collection  of 
positions  vAiich  eure  similau:  in  task  oaitent  eind  vhich  differ  frcm  other 
jobs;  a  position  is  the  work  dcxxe  by  an  individual.  Job  content,  however, 
is  described  in  terms  of  the  Tllfe  (Task  Training  Modules)  produced  by  the 


TCS.  i^nopriate  association  indices  have  been  developed  to  describe  the 
relaticaiship  between  a  given  job  and  each  individued  TIM. 


The  second  major  characteristic  is  the  set  of  training  states 
cissociated  with  eui  AFS.  A  training  state  is  a  particular  exaitple  of  a 
training  setting;  for  example,  a  particular  FTD.  Thus,  a  particular 
stream  of  successive  jcb  emd  training  states  will  represent  an 
AFS-specific  typical  career  path  fran  Basic  Military  Training  (EMT) , 
throu^  the  first  resident  techniccil  school  (in  most  cases) ,  then  throu^ 
the  first  job  assignment  and  additional  training  required  for  that  jc±). 
Fincilly,  subsequent  jobs  and  cissociated  training  eire  represented  until 
s^jaration  frcm  the  Air  Force  occurs.  Because  U&T  patterns  must  be  easily 
presented  to  and  understood  by  managers,  seme  simplifying  principles  and 
cissumptions  cure  required.  For  example,  atypical  positions  may  be  grouped 
("other  jobs")  or  deleted  frcm  the  fin^  models  to  avoid  the  "cognitive 
overload"  that  could  arise  frero  consideration  of  overelaborated  models. 

Thus,  the  major  product  of  this  cotponent  is  a  representation  of  the 
current  U&T  pattern  in  an  AFS.  The  r^resentation  is  developed  in 
sufficient  detail  to  allow  accurate  functioning  of  the  integrated  TDS. 
Initicil  inputs  include  data  from  the  most  recent  OS  for  the  APS,  the 
results  of  the  subsequent  re-aneilysis  and  new  emcilyses  performed  on  those 
data,  information  concerning  training  courses,  personnel  and  manpower 
data,  and  information  gathered  from  subject-matter  experts  (SMEs)  eind  job 
incumbents. 

1.3.2  Alternative  U&T  Pattern  Oonaaonent 

Alternative  U&T  patterns  are  described  within  the  same  framework  as 
the  current  U&T  pattern,  to  allcw  maximum  comparability  in  determining 
managers'  preferences.  Alternative  patterns  include  logical  possible 
changes  to  the  current  pattern;  for  example,  the  provision  of  job  training 
in  settings  other  than  those  xxw  used.  Such  changes  might  reflect  a 
theoreticedly  determined  optimizaticxi  of  resource  usage,  such  eis  a 
minimizaticn  of  training  costs.  Or,  such  changes  can  reflect  Air  Force 
management  plans,  expectations,  and  reccmmendaticaTS  about  the  future 
structure  of  an  AFS.  For  purposes  of  the  HJS,  these  alternatives  will  be 
both  described  verbally  and  expressed  in  the  framework  of  a  mathematical 
model.  Any  actual  optimization  operations  would  make  use  of  data  from 
other  subsystems  as  well,  and  therefore  be  carried  out  by  the  lOS. 

1.3.3  Manaaeament  Preferences  Oomponent 

In  this  oomponent,  the  various  U&T  patterns  eire  presented  to  key 
managers  ooncemed  with  the  specialty.  Such  managers  are  given  the 
opportunity  to  express  their  preferences  for  the  patterns,  beised  primarily 
vpon  OOToems  of  preparing  AFS  incumbents  to  efficiently  perform  the  work 
required  by  the  Air  Force.  Ideally,  other  considerations  such  as  costs 
would  not  affect  their  preferences;  such  concerns  are  the  donain  of  the 
RCS.  In  practice,  however,  it  may  not  be  possible  to  restrict  the 
criteria  cxi  vJiich  preferences  are  based.  In  any  event,  one  purpose  of 
this  component  is  to  edlow  comparison  of  preference  data  across  managerial 
groups.  For  example,  the  preferences  of  functicxTal  managers  mi<^t  be  very 
different  fromi  those  of  personnel  or  training  managers.  Identification 
eind  resolution  of  such  differences  is  a  primary  purpose  of  U&T  Workshops 
and  TFT  meetings. 


2.0  RESEARCH  AND  nFVFTnPWFOT  OF  THE 
FIEID  irriLIZATICN  SUBSYSTEM 


2.1  The  Oirrent  Utilization  and  Training  Pattern  (Sarponent 
2.1.1  Air  Force  Jda  Typing 

Ihe  Air  Forcje  method  of  job  analysis,  or  jcb  typing,  hcis  evolved  and 
iirproved  over  a  period  of  some  25  years  or  more  (Christal,  1974) .  Perhaps 
the  best  introduction  to  this  subject  for  an  interested  reader  is  Archer 
(1966) ,  vrtiich  will  acquaint  one  with  the  basic  rationale  and  procedures; 
althoiK^  many  more  sv^iporting  materials  are  available  today  than  vhen  it 
was  written,  the  techniques  of  c^jerational  jcb  typing  have  not  changed 
over  the  years.  Fundamentcilly  different  cLLtematives  have  begun  to  appecir 
(see,  for  exaitple,  Mitchell  &  Ehcilen,  1985) ,  but  these  etre  still  in  the 
research  stage. 


Vi 


Basically,  ein  AFS  is  cono^3tually  ccnprised  of  a  set  of  tasks  vhich 
describe  the  work  done  by  speciality  menbers.  A  group  of  very 
knowledgeable  senior  meirbers  of  the  speciality  eme  periodicailly  charged 
with  reviewing  and  revising  a  task  list  produced  by  the  Oca^jationail 
Meaisurement  Center  (CMC) .  (There  are  typicailly  several  hundred  tasks  in 
the  finail  task  list.)  This  task  list  is  included  in  am  OS,  aind 
distributed  by  the  Occi?»tionail  Survey  Branch  of  CMC.  A  large,  often  full 
census  sample  of  appropriate  airmen  receives  the  survey,  vhich  they 
complete  by  checking  in  the  survey  booklet  each  task  they  perform  in  their 
present  assignment,  and  rating  on  a  nine-point  scale  the  amcfunt  of  jcb 
time  they  spend  on  the  task.  This  time  rating  is  standardized  or 
converted  into  an  index  resembling  a  percent.  Analysis  of  the  similarity 
of  work  done  by  amy  two  or  more  individuails  can  then  proceed  according  to 
either  the  sheer  number  of  taisks  they  perform  in  octnmon  or  according  to 
the  "percent"  of  work  time  they  spend  on  tasks  they  perform  in  cxmmon. 
Similarity  indices  are  cxmputed  by  a  hierarchicail  clustering  ailgorithm  due 
initially  to  Waud  (1963) .  The  adgorithm  begins  by  clustering  the  most 
similar  people  (or  positions) ,  then  iteratively  draws  additionail  pecple 
into  the  various  groips  and  forms  new  grcups  as  necessary.  In  effect,  the 
initial  grcxps  grew  in  size  and  merge  with  one  another  until  a  final 
cluster  comprises  the  entire  sample.  The  overall  similaurity  of  groip 
menbers  to  one  another  decreases  gradually  as  the  grotps  grow  in  size.  A 
human  anadyst  must  examine  a  hierarchical  clustering  diagram  aind  decide 
just  vhich  clusters'  members  aore  similar  enou^  to  each  other  and 
different  encu^  frcmi  those  of  nei^boring  clusters  to  be  considered 
identifiable  jobs. 


There  aure  sente  potentiail  pitfalls  inherent  in  this  process.  First,  it 
is  hi(^iLy  subjective;  adthough  the  clusters  and  similaurity  indices  will 
always  be  the  same  (other  things,  sxxb  as  starter  grxxp  size,  being 
equal) ,  it  is  not  impossible  for  two  different  anadysts  to  identify 
scntevbat  different  groups  as  jobs.  Second,  the  algorithm  used  often  fails 
to  place  5%-20%  of  AFS  members  in  any  cluster  until  the  within-grxxp 
similarity  is  so  lew  that  no  ejperienoed  anadyst  would  ocMisider  it  a  job. 
This  is  not  intended  as  criticism.  Most  people  familiar  with  the 


job-typing  prooess  eigree  that  it  is  valuable,  althcu^  seme  utproveraents 
in  the  technology  are  needed.  Several  new  methods  are  being  developed  and 
evaluated  (see  Mitchell  &  Ftalen,  1985;  Ihalen,  Mitchell,  &  Stal^, 

1987;  Rialen,  Staley,  &  Mitchell,  1987). 


2.1.2  First-term  Versus  Career  Job  Types 

We  began  the  job-typing  effort  by  exploring  the  issue  of  overlap 
between  jobs  held  by  first-term  and  career  ainnmi.  It  was  cxxisidered 
possible  that  they  perform  fadrly  distinct  technical  jobs  in  some 
speciadties,  with  little  overlap  between  first-term  and  career  members' 
taste  or  the  skills  and  knowledge  required  for  those  taste.  Ihere  are 
obvious  inplications  for  the  RJS  if  this  is  so  for  any  of  the  prototype 
IK  ^lecialties. 

The  typical  job-typing  exercise  begins  by  creating  a  file  that 
contains  adl  the  data  from  the  OS.  Since  each  ICS  qjecialty  had  been  the 
subject  of  a  recent  stuc^  by  CMC,  a  file  for  each  already  existed,  stored 
on  auxhived  tapes.  Die  Oenprehensive  Ooexpationad  Data  Anadysis  Programs 
(OOCAP)  ^stem,  v^ch  is  avadlable  on  the  AFMRL's  Sperry-Wiivac  oemputer, 
contains  various  vitility  programs  vhich  can  retrieve  files  from  archives 
and  prepare  them  for  further  analysis. 

Once  these  files  are  rea<^,  additional  OCOAP  routines  automate  the 
job-typing  prooess  ip  to  the  point  of  producing  a  tree-lite  diaigram  of  the 
clustering  results.  Some  of  the  programs  produce  additionad  files 
containing  the  results  of  their  calculations  in  a  form  suitable  for  inpit 
to  further  programs.  Ihe  anadyst  has  both  the  products  from  these 
programs  and  additionad  optional  programs  to  work  with  in  determining  the 
job  types.  D»  present  effort  followed  this  basic  procedure,  exo^  that 
for  AESs  811XX  (Security  Police)  and  328X4  (Avionic  Inertial  and  Rad^ 
Guidance  Systems)  the  prooess  was  r^ieated  on  separate  files.  One  file  in 
each  speciadty  contained  data  for  first-term  personnel,  who  were  defined 
as  those  with  vp  to  48  months  Total  Active  Federal  Military  Service 
(TAfMS) .  Die  other  file  oontadned  data  for  career  personnel,  who  had 
TAFM5  of  greater  than  48  months. 

2.1.3  Results  for  AFSs  328X4  and  811XX 

Die  results  of  job  typing  were  documented  in  informal  comnonications 
throug^iout  the  effort,  and  finally  in  detail  as  a  hard-copy  contract  data 
deliverable.  Die  beisic  result  is  that  the  job  types  identified  by  the  CMC 
analyst  and  described  in  the  Ooapational  Survey  Report  (OSR)  for  each 
q3eci2ilty  were  adopted  with  few  changes  for  use  in  the  prototype  IDS. 
Althou^  there  are  some  distinctly  first-term  or  career  job  types,  the 
majority  of  jobs  are  oerprised  of  a  mixture  of  the  two.  S^icurate 
clusterings  were  not  found  to  produce  substantieilly  different  sets  of 
jobs.  In  a  few  cases  new  jobs  could  have  been  created,  but  there  was  no 
reason  to  believe  that  this  approach  would  offer  any  eidvcuitages  over  the 
CMC  jobs  for  the  purposes  of  the  IDS. 

2. 1.3.1  AFS  328X4.  In  APS  328X4,  the  sep^tion  of  fi^-berm 
and  career  personnel  did  not  edter  the  basic  oenposition  of  the  job 
types.  Diere  were  a  few  jobs  that  were  conprised  ocnpletely  or  eOmost 
conpletely  of  either  first-term  or  career  personnel,  so  some  gixxps  were 
evident  in  one  clustering  diagram  and  not  the  other.  Die  career-only  jobs 
were  all  senior  supervisory  and  management  positions.  Die  first-term-only 


jobs  tended  to  be  small  groc^,  the  manbers  of  vdiich  were  very  new  to  the 
career  field  and  performed  mainly  very  siitple  maintenance  functions.  For 
the  most  part,  jcbs  ccnsisted  of  fairly  equal  numbers  of  career  and 
first-term  peisonnel  v4io  r^xsrted  doing  very  similar  sorts  of  work.  None 
of  these  findings  constituted  a  surprise,  however;  this  sort  of 
information  emerges  from  a  routine  job  eincd.ysis  vAiich  does  not  i^lit  the 
specicilty.  The  i^lit,  however,  did  cause  sc^^erficial  disn;$]tion  of  the 
clustering  patterns,  with  lingering  s^>aration  of  seme  people  vho  came 
together  during  early  iterations  of  the  vhole-sanple  clustering.  But  the 
reasons  for  separation  were  not  substantial  because  the  differences  that 
k^it  them  apart  were  merely  subtle  variations  in  performance  of  the  same 
basic  groeps  of  tasks  or  the  order  in  vAiich  they  linked  early  in  the 
hierarchical  process. 

The  analysis  of  AFS  328X4  revealed  a  gradual  career  progression  from 
the  very  simple  medntenanoe  tasks  assigned  to  some  very  new  personnel, 
through  a  long  intermediate  career  that  involves  primarily  challenging 
technical  work  with  gradually  increeising  sipervisory  and  management 
opportunities,  culminating  in  purely  sipervisory/managanent  activities  for 
very  senior  perscminel.  Even  so,  most  members  (including  qiiite  senior 
members)  of  the  ^)ecialty  remain  involved  in  technical  work. 


2. 1.3. 2  AES  ailXX.  A  subcontractor  (Research  Triangle 
Institute)  performed  the  preliminary  analysis  of  the  811XX 

specialty.  The  findings  closely  paralleled  those  of  the  1984  06R.  For 
example,  some  35  job  types  were  identified,  vhereas  the  06R  identified 
34.  A  more  detailed  analysis  by  McDonnell  Douglas  personnel  resulted  in 
the  identificatiem  of  41  first-term  jobs.  De^ite  epparmit  disparities, 
close  examinaticxi  shewed  that  seme  of  these  job  types  were  equivalent,  and 
that  only  minor  differences  in  task  ordering  characterized  the  grtxps. 
Equivalent  jobs  were  therefore  assigned  the  same  job  title  in  the  fin^l 
analysis,  and  the  06R  jobs  were  used  with  only  minor  changes.  As  witki.  AFS 
328X4,  the  career  field  can  be  described  overall  as  involving  a  set  of 
very  simple  first  jobs,  a  set  of  purely  sipervisory/managerial  jobs 
performed  1;^  very  senior  perscxinel,  and  a  large  set  of  intermediate  jobs 
within  vhich  gradueil  increases  in  sipervisory  opportunities  are  apparent. 
This  simplifies  the  picture  semevhat;  additional  ocmplications  result  from 
the  fact  that  AFS  811XX  actually  encompasses  three  different  career 
fields.  This  results,  for  example,  in  restrictions  on  movement  between 
811X2A  (Military  Working  Dog  Handlers)  and  the  rest  of  811X2  (law 
Enforcement) .  Some  81IX2A  members  believe  that  their  sipiervisory 
opportunities  are  constrained  in  conparison  to  those  of  811X2  members. 

The  irpact  of  these  issues  on  the  prototype  IDS,  however,  appears  minimal. 

2.1.4  AlSs  305X4  (Electronic  Ooncuter  and  Switching  Systems^  and  423X1 
(Aircraft  EnvircaTmental  Systems) . 

The  job-typing  method  employed  for  AFSs  305X4  and  423X1  was  semefchat 
different  than  that  \jsed  for  the  other  specialties.  One  difference  is  of 
little  interest  for  present  purposes,  but  should  be  mentioned  in  passing. 

Ttiese  specialities  were  analyzed  using  tiie  new  version  of  CXXAP, 
written  in  ASCII  FOE^TRAN,  vhich  is  new  being  tested  by  AFHRL  and  CMC. 

This  is  an  ipdated  and  expanded  version  of  tt^  older  Fielddata  versicxi  of 
OCXAP  (Ihaaen,  Weissmuller,  &  Staley,  1985) .  The  purpose  of  the  new 
system  is  the  same,  but  in  many  respects  it  is  more  streamlined  auid  easier 


to  vise  than  the  older  version.  Ihere  are  also  new  program  names,  often 
for  programs  vftiich  ocntoine  more  than  one  old  version  program  into  a  single 
st^.  There  are  aviso  atdditional  ca^iabilities,  but,  by  the  same  token,  the 
older  version  was  also  expanding  constantly.  In  any  event,  the  sequence 
of  job-typing  events  was  logically  the  same  vmder  either  version.  There 
were  some  problems  eilong  the  way,  usually  involving  "tugs"  found  in  new, 
relatively  untested  programs,  or  else  involving  revisiOTis  to  programs  and 
documentation  in  the  middle  of  some  particular  effort.  Working  with  two 
systems  presented  certain  problems,  adthou^  the  new  system  represents  a 
considerable  iitprovement  over  the  old  in  mavry  respects. 


The  other  difference  is  more  substantiail.  Whereas  the  analysis  for 
811XX  and  328X4  consisted  of  producing  and  working  with  s^iarate  diagraite 
for  first-term  and  cavteer  airmen,  the  lack  of  interesting  differences 
between  the  CMC  job  types  and  those  identified  in  the  split  samples 
suggested  that  a  sinpler  approach  was  probably  acceptable. 

The  two  additional  AESs  were  analyzed  by  producing  a  single  clustering 
solution  and  diagram  for  the  entire  specialty,  then  using  CODAP  prograns 
to  ^lit  each  CMC  job  type  into  first-term  and  career  grovps.  Additional 
CODAP  programs  produced  duty-  and  task-level  job  descriptions  of  each 
split  job.  As  before,  only  very  sli^t  differences  were  found  between 
first-term  and  career  airmen  in  any  giv^  job  occvpied  by  both.  The 
differences  were  never  large  enou^  to  indicate  that  the  two  should  be 
considered  distinct  job  ^pes. 

Subsequent  refinement  of  423X1  jobs  resulted  in  a  large  CMC  job  type 
being  split  into  three  pieces,  of  viiich  two  were  small  job  types  and  the 
third  was  nearly  as  large  as  the  original.  All  the  jobs  in  the  305X4  OSR 
were  adopted  with  no  changes.  This  was  necessary  even  thcu^  we  soon 
discovered  that  najor  changes  to  the  structure  of  the  305X4  ^lecialty  were 
planned  in  the  near  future.  These  changes  involve,  among  other  things, 
creation  of  some  new  "shredouts,*  deletion  of  some  old  shreds,  and  turning 
the  work  of  some  Air  Force  personnel  over  to  civilian  personnel  (to 
parties  as  diverse  in  nature  as  the  Federcd  Aviation  Administration,  and 
even  the  West  German  tel^hone  ocnpai^  in  the  case  of  some  overseas 
switching  systems  maintenance) . 

2.1.5  Dynamic  Ctoss-KPAITT  Analysis 

Jobs  are  cxily  one  coapcxient  of  the  U&T  pattern.  A  conccmitant  effort 
was  required  to  identify  the  training  states  (e.g. ,  resident  courses. 

Field  Training  Detachments  or  FTDs,  Career  Development  Courses  or  CDCs, 
Qn-the-Job  Training  or  QJT,  etc.)  that  are  needed  to  pr^are  people  for 
their  jobs,  and  determining  the  likelihood  that  training  will  be  received 
as  it  is  needed.  The  goal  of  another  related  effort  was  to  determine  the 
pattern  of  job  flow;  that  is,  the  "transition  probability"  that 
incunbents  in  any  given  job  would  be  assigned  next  to  any  other  given  job 
in  the  ^jecialty. 


One  possible  starting  point  for  determining  transition  probabilities 
was  so-called  "(tynamic  cross-KPAIHing. "  The  procedure  involves 
idaitifying  people  vho  participated  in  two  (or,  in  theory,  more) 
consecutive  OS  efforts.  Th^  may  be  identified  by  backgrcxind  information, 
such  as  social  security  nuiiber.  Subsequent  time  1-tlme  2  job  conparisons 
are  made  according  to  the  "KPATH"  nunber,  an  index  assigned  by  the 


clxjstering  procedure  vAiich  indicates  the  point  at  vAiich  an  individueO. 
entered  a  cluster.  Ihis  index,  obviously,  lends  its  name  to  the 
procedure. 

The  next  st^  involves  determining  which  job  each  person  was  in  during 
each  survey  period.  The  expected  result  would  be  a  matrix  of  job-to-job 
transiticxi  probabilities  that  can  be  refined  hy  additional  efforts. 

We  performed  a  cross-KPAIH  aneilysis  on  three  specialties  (under  an 
agreement  with  AFHKL  staff,  we  did  not  plan  to  try  zdl  possible  ways  of 
building  U&T  patterns  for  each  speciedty) ;  the  AESs  selected  for 
cross-KPAIH  analysis  included  423X1,  305X4,  and  328X4.  The  resulting 
matrices  and  additioneil  issues  were  briefed  at  ana  of  the  TDS  progress 
reviews. 

The  cross-KPAIH  effort  did  produce  reasonable-looking  matrices. 
However,  a  number  of  practical  and  theoretical  problems  limited  the 
usefulness  of,  or  at  leeist  the  level  of  confidence  one  could  have  in  the 
method.  There  are  three  particular  problems,  listed  here  in  the  order  of 
severity. 

First,  the  sample  sizes  may  be  small,  and  the  estimates  in  seme  cells 
can  be  based  on  very  few  cases.  This  problem  can  be  aggravated  by  the 
fact  that  seme  OS  samples  cenprise  a  relatively  small  part  of  the 
specialty,  as  was  the  case  with  811XX.  That,  and  the  relatively  lew 
retenticai  level  in  that  specialty  after  the  first  term,  reduce  the 
likelihood  that  cross-KPAIH  anedysis  will  be  based  on  a  sample  that  is 
sufficiently  large  or  r^resentative. 

Seccaid,  even  with  a  large  sample  one  cannot  be  certain  that  any 
particular  cell  entry  represents  single-step  job-to-job  transitiOTis.  As  a 
general  rule,  several  years  intervene  between  OS  efforts,  althou^  this 
can  vary  because  CMC  conducts  studies  mainly  at  t2)e  request  of  AES 
managers.  Data  discussed  below  suggest  that  the  average  job  incumbency 
period  in  many  ^)ecialties  is  ocsTsiderably  shorter  than  the  typical  time 
between  OS  efforts.  The  point  is  that  ary  given  individued  in  a  sample 
could  represent  one,  two,  or  even  three  (and  even  more,  in  rare  cases)  job 
transitiemis.  When  \ised  in  a  dynamic  model,  such  informaticxi  mi^t  be 
misleading  in  important  ways. 

Third,  the  jobs  are  not  edways  easy  to  cenpare  between  two  surveys. 
This  results  in  part  from  the  inherent  subjectivity  of  the  jcb-l^ing 
process,  as  menticMied  earlier.  It  is  possible  that  two  analysts  will 
identify  soroevhat  different  jobs  vhen  analyzing  the  same  survey  data.  The 
overlap  is  likely  to  decrease  vhen  severed  years  intervene  between  two 
surveys,  and  additional  variability  results  from  actual,  substantial 
changes  in  the  nature,  training,  mission,  and  equipment  in  the  career 
field,  quite  apart  from  the  analyst's  peroepticxis  and  decisicxis.  Finally, 
policy  changes  at  CMC  change  from  time  to  time.  The  level  of  ^jecificity 
at  vhich  an  analyst  works  (i.e.,  the  degree  of  difference  that  must  exist 
between  groeps  for  them  to  be  called  separate  jobs)  is  in  part  a  matter  of 
current  policy  and  practices. 

The  cross-KPAIH  analysis  of  AES  328X4  can  serve  as  a  case  in  point. 

The  two  most  recent  06Rs  for  this  career  field  appeared  about  4  years 
apart,  in  1978  and  1982.  The  1978  report  identified  10  jobs  in  the  career 


field,  and  the  1982  report  Id^ified  26.  At  one  point,  we  consulted  with 
a  very  experienced  SME,  a  recently  retired  328X4,  vAio  provided  us  with  a 
mapping  of  the  later  job  l^ing  onto  the  earlier  one.  Ihis  mi^t  have 
resolved  the  problem  if  the  mzpping  had  been  smooth  (for  exanple,  if  two 
or  three  1982  jobs  had  meqpped  neatly  onto  each  single  1978  job) . 
Uhfortunately,  the  mapping  this  SME  produced  was  not  so  neat;  for  exanple, 
part  of  a  1982  job  mic^t  map  onto  one  1978  job,  another  part  mic^t  map 
onto  another  job,  a  third  part  mi^t  not  map  onto  any  1978  jobs,  etc.  At 
the  same  time,  a  single  1978  job  mi^t  have  parts  of  several  1982  jobs 
mapped  onto  it.  In  short,  there  seemed  no  easy  way  to  determine  what  the 
transition  probabilities  meant,  at  least  not  without  determining  how  to 
proportion  them  among  various  alternatives.  Ihis  in  itself  would  have 
been  a  very  subjective  procedure. 

Ihe  cross-KPA!IH  analysis  was  not  meant  to  produce  final  transition 
matrices,  but  to  produce  a  helpful  beginning  point  for  further 
refinement.  Ihe  same  SME  who  translated  jobs  between  OSRs  was  also 
working  on  the  SME-defined  U&T  pattern  effort  described  in  Section  2.1.6 
of  this  paper.  He  was  offered  the  cross-KPAlH  matrix  as  an  aid  but  chose 
not  to  tise  it.  The  point  is  that,  in  this  case  (i.e. ,  this  SME,  this 
career  field,  and  this  pair  of  job  l^pe  sets) ,  the  cross-KPATH  product  was 
not  even  considered  a  good  starting  point.  Althcu^  we  performed 
ocnparable  analyses  on  two  other  specialties,  we  did  not  have  another 
opportunity  to  map  the  results  of  one  analysis  onto  another. 

2.1.6  SME  Jidaroents 

We  e9q)lored  a  second  method  of  devising  U&T  patterns.  This  involved 
sinply  working  with  a  very  knowledgeable  SMi  or  panel  of  SMEs,  vhose 
esqierience  mi^it  allcw  them  to  draw  a  schematic  (or  do  scmething  equally 
suitable)  of  career  field  job  flow  and  describe  the  training  (as  it  is 
presently  given,  not  the  ideal)  that  prepares  people  for  jobs.  In  other 
words,  perhe^  SMBs  could  si;pply  ed.1  that  was  required  at  a  suitable  level 
of  accuracy. 

By  agreement  with  AIHRL  staff,  we  tested  this  method  on  each  of  the 
proto^pe  IDS  qjecialties  except  811XX.  There  were  several  different 
tests  for  328X4  and  305X4,  both  with  consultants  and  with  SMEs  officiedly 
provided  by  AfHRL.  Most  of  the  official  testing  effort  took  placse  during 
trips  to  Sheppard  and  Tinker  AFBs  during  the  sunmer  of  1986,  and  to  Air 
Force  ^»oe  Ocnmand  Headquarters  in  Colorado  Springs  in  the  fall  of  1986. 

The  results  of  these  efforts  were  submitted  to  AFKkL  as  Preliminary 
U&T  Models.  In  fairness,  the  fundamental  job-flcw  patterns  produced  by 
the  best  SMEs  were  not  b^  (SMEs*  abilities  with  this  excerclse  varied 
considerably,  although  we  always  found  a  few  vho  epparmntly  did  it  well) . 
We  provided  SMEs  with  information  we  thou^t  micht  help  (including,  as  has 
already  been  said,  the  cross-KPATH  results) .  SMEs  found  much  of  the 
informaticxi  avedlable  in  an  06R  helpful,  especially  that  concerning  such 
things  as  average  grade  and  TAFIG  of  job  type  menbers,  the  most  frequently 
performed  tasks,  additicml  narrative  job  descriptions,  and  the  like. 

They  were  able  to  use  this  Informaticxi  in  oonjvinctlon  with  their  cwn 
knowledge  (for  exanple,  about  how  assignments  typically  work  in  their 
i^ieciadty)  to  produce  job  flew  diagrams  that  were  reasonably  accurate 
cenpared  to  those  produced  by  our  detailed  06  data  anedysls  (see  Section 
2.1.7). 


HcMever,  sane  general  findings  serve  to  limit  the  usefulness  of  this 
method  for  determining  U&T  patterns.  First,  even  the  most  eiqierienoed 
people  had  trouble  with  the  job-training  relatioiships  that,  along  with 
job  flews,  are  very  important  for  accurate  U&T  patterns.  Most  simply  said 
they  did  not  knew  enou^  about,  for  example,  the  current  or  recent  FIDs  to 
determine  the  relatioiship  between  them  and  jobs  with  oanfidenoe.  Diey 
certadnly  did  not  feel  ccaifident  about  estimating  the  percentage  of  job 
incumbents  viio,  in  connection  with  their  job,  receive  any  given  FID  or 
resident  school  training  state. 

Ihe  second  issue  is  a  bit  more  subtle.  Althou^  the  job  flew  patterns 
SMEs  produced  were  fairly  accurate,  they  were  a  bit  oversimplified.  For 
example,  jobs  were  seen  as  available  in  too  narrow  a  time  window;  SMEs 
tended  to  r^resent  jobs  too  discretely  as  first-term  or  seoaid-term  or 
third-term.  The  effects  of  this  can  accuracy  are  probably  more  severe  with 
some  of  the  very  large  job  grotps  than  with  some  of  the  smaller  ones,  but 
SMEs  did  not  seem  generally  very  ser^itive  to  the  stated  size  of  the  job 
grofips.  They  cilso  failed  to  include  some  jobs  in  their  models,  perhaps 
simply  overlooking  them.  Ihey  were  sometimes  critical,  because  a  job  they 
believed  ^ould  be  r^resented  was  not.  All  this  is  understandable,  but 
raises  questions  about  the  suitability  of  this  method.  It  is  possible 
that  very  detailed  job  typings  will  eventually  be  judged  unsuitable  for 
the  TD6;  simplicity  in  this  regard  already  has  proponents.  Even  so,  some 
additianeLL  method  would  still  be  needed  to  determine  job-training 
associations. 


2.1.7  Additional  Surveying  and  OS  Data  Analyses 

Early  in  this  project,  a  consultant  hired  liy  AEETOL  to  review  new  TDS 
research  and  development  plans,  stated  in  his  report  that  he  did  not  see 
any  way  of  getting  accurate  job-training  association  estimates  without 
considerable  (and  at  that  time  vmplanned)  additioial  surv^ing  (Moore, 
1984) .  It  appears  that  he  was  probably  ri^t. 

Althoucfi  the  information  available  from  OS  data  is  generally 
excellent,  the  06  for  most  specicilties  lists  very  few  of  the  courses 
actually  available  to  specialty  members.  Even  for  these  courses,  there  is 
no  information  about  vhen  the  course  was  taken;  therefore,  a  particular 
course  cannot  be  confidently  associated  with  the  member's  current  job. 
(Ihese  cire  not  inherent  or  necessary  problems;  the  06  could  be  devised  so 
as  to  provide  this  and  other  information.)  cross-KPAIH  aneilysis  never 
promised  to  provide  job-training  association  data.  On  the  other  hand,  it 
was  diseppointing  to  find  most  SMEs  unwilling  even  to  provide  guesses 
about  these  associations. 

Another  possibility  involved  analysis  of  individual  airmen's  personnel 
records,  but  there  were  several  practiced,  problems.  For  example,  edtheu^ 
the  recent  training  history  would  be  reliable,  it  would  still  be  difficult 
to  associate  training  states  to  jobs  becaixse  persomel  records  carry  only 
very  generic  job  titles.  Mapping  these  onto  CMC  job  type  titles  would 
involve  yet  another  series  of  judgments  by  an  aneilyst  or  8ME.  In 
addition,  there  were  problems  with  access  to  personnel  records. 

2. 1.7.1  Job  and  Training  History  6urvevs.  We  devised  and 
field-tested  a  Job  and  Training  History  8urvey  (J&UB) ,  to  gather 


information  on  c^namic  job  flew  and  job-training  association  information 
unavailable  frem  other  sources.  Uie  original  agreement  with  AFHRL  was 
that  an  additional  surveying  effort  wculd  cover  oily  AFSs  328X4  and 
811XX.  the  feQ.1  of  1986,  however,  none  of  the  other  methods  had 
produced  UiT  patterns  that  met  expectations.  Uie  additicsial  surveying  was 
extended  to  AFSs  305X4  and  423X1,  edthouc^  project  time  was  very  limited 
by  that  time.  Further  details  aixxit  the  results  of  this  effort  were 
provided  to  APHRL  as  Data  emd/or  Analysis  Suimnary  -  Current  U&T  Patterns, 
Task  II. 

What  the  J&THS  requested  depends  on  the  ^ecialty.  Ihe  survey  for  AFS 
328X4  began  with  a  copy  of  the  standard  baokground  inforroaticxi  page  and 
Privacy  Act  Statement  incliided  in  06  paokages.  Ihe  first  section 
requested  information  on  training  history,  asking  specifically  vhether  and 
when  the  re^xxident  had  taken  various  FID  and  resident  courses, 
inter-service  courses.  Mobile  Training  Team  courses,  professional  military 
education  (IME)  courses,  etc.  Ihe  section  contained  the  specific  names 
and  nuiibers  of  as  many  courses  as  possible.  Ihis  information  was  gathered 
mainly  from  new  and  old  volumes  of  Air  Force  Regulation  (AIR)  50-5,  IKAF 
Formal  Training  Schools,  but  other  sources,  incl\iding  old  and  more  recent 
06  forms  and  SME  input,  proved  helpful.  Generous  ^)aoe  was  provided  for 
fill-ins,  along  with  instructions  asking  for  any  other  courses  not 
inclxided  in  the  list. 

Ihe  second  section  requested  information  about  jobs.  Each  job  type 
was  described  by  a  title,  brief  narrative,  and  short  list  of  tasks 
typically  perfemed.  Participants  were  to  read  these  descariptiens  and 
re^xxid  with  dates  if,  in  their  opinion,  they  had  ever  ooexpied  the  job. 
Ihey  were  instructed  to  pioik  one  best-fitting  job  for  each  time  period 
(except  for  filling  in  any  periods  of  ^)ecial  assignments,  of  ^Aiich 
examples  were  given) ,  and  leave  no  time  gaps  unaccounted  for. 

Additional  sections  requested  information  about  the  participant's 
major  oenmand  (MATOCM) ,  aircraft  worked  on,  emd  systems  worked  on,  for 
each  job  selecticxi  made  in  the  previous  section. 

A  first  form  of  this  survey  was  pilot-tested  with  the  help  of  AFS 
328X4  personnel  at  Langley  AFB  in  August  1985.  It  was  revised  eis  a 
result  of  their  oonments,  and  underwent  additional  revision  as  a  result  of 
suggestions  by  AIRRL  st^lff  menbers.  It  was  administered  in  the  field  in 
the  f2dl  of  1986. 

Ihe  survey  for  AFS  811XX  differed  scroefchat  from  that  for  328X4.  Ihe 
original  form  was  very  similar,  but  pilot-testing  at  Scott  AFB  in  January 
1986  shewed  that  considerable  revisicai  was  needed.  One  revision  was  to 
include  Educational  Subject  Block  Indices  (ESBis)  as  a  primary  training 
mode  for  this  AFS.  Ihe  main  revisicxi,  however,  resulted  from  the 
resistance  of  811XX  personnel  at  Scott  to  picking  a  single  job  type  for 
ary  given  time  period.  Ihey  insisted  that  their  jobs  were  broad, 
including  mary  short-^term  assignments  to  special  functions,  and  daily  jobs 
vhich  cut  across  many  of  the  job  types  identified  in  tha  CCR.  For  this 
reason,  on  the  final  J&ne  respondents  were  asked  to  rarrk  a  long  list  of 
811XX  functional  areas  according  to  the  amcuiTt  of  time  ^)ent  )y  each  in 
their  present  assignment,  previous  assignment,  etc. ,  instead  of  being 
asked  to  pick  jobs.  We  were  hopeful  that  06  job  descripticxis  could  then 


be  related  to  functional  area  ranking  patterns  (but  met  with  little 
success  in  doing  so) .  This  survey  was  cilso  in  the  field  during  the  fall 
of  1986. 

The  J&THS  for  AESs  305X4  was  similar  to  the  328X4  survey  with  regard 
to  asking  respondents  to  select  OS  jcbs  for  each  assignment  period.  The 
one  for  423X1  was  like  that  for  811XX  in  this  respect,  requesting  rankings 
of  functional  areas  worked  in  each  assignment.  This  approach  seemed  best 
in  li^t  of  the  unconventional  hcanogeneitY  of  423X1  indicated  in  the  OSR. 
We  thcu^t  it  mi^t  be  possible  to  lase  data  on  423X1  functional  areas,  if 
necessaiYf  for  additioned.  breakdown  of  the  speciedty  and  for  the 
job-training  ancilysis.  We  were  unable  to  pilot-test  these  surveys  to 
determine  if  this  was  the  best  way  to  gather  job  information  in  these 
specicilties,  due  to  the  short  period  of  time  remaining  in  the  project. 
Finally,  these  surveys  were  somevhat  abridged,  relative  to  those  for  328X4 
and  811XX.  This  was  because  they  were  intend^  cts  a  quick-response  effort 
for  the  main  purpose  of  gathering  seme  then-still-u»,available  training 
data. 

We  were  unable  to  ancilyze  and  use  the  full  ccnplement  of  information 
which  resulted  from  this  surveying  effort,  partly  because  of  the  unantici- 
pdted  additional  surveying  and  peu±ly  because  surveys  arrived  in  the  field 
much  later  than  originally  anticipated.  It  is  possible  that  the 
additioncil  information  available  from  these  surveys  will  be  of  iise  in  the 
future  on  this  or  another  project.  As  it  is,  data  from  the  J&THS  were 
used  in  the  prototype  TDS  mainly  to  estimate  the  likelihood  that  a  job 
incumbent  will  receive  training  in  a  particular  training  state.  In  seme 
cases,  additional  information  from  the  survey  was  used  to  estimate  the 
point  in  an  average  career  vhen  certain  training  is  given.  For  the  811XX 
pattern,  information  from  the  survey  was  also  used  to  estimate  CDC  course 
completion  points,  FME  course  sequence  and  tindng,  ESBIs  completed,  and 
the  duration  of  cin  average  first  cissignment,  second  cissignment,  etc. 

2. 1.7. 2  Additional  OS  Data  Analyses.  In  the  end,  the  OS  remains 
the  best  source  of  much  information  about  the  current  U&T  pattern.  The 
data  are  often  based  on  an  effort  which  affords  everyone  in  the  career 
field  the  opportunity  to  respond  or  samples  sizeable  numbers  of  Icurge 
AF5s.  It  is  also  easy  to  use;  the  ODDAP  program  called  FHTVAR  eeisily 
extracts  individuals'  OS  data  from  the  Case  Data  File,  and  convenient 
data-aneilysis  software  systems  such  as  the  Statisticed  Package  for  the 
Social  Sciences  (SPSS)  may  be  used  for  manipulating  and  analyzing  the 
information.  The  biggest  dravhacks  (for  present  purposes)  include  the 
lack  of  dynamic  job-flow  information,  the  relative  incompleteness  of  the 
training  information  in  many  cases,  and  the  unavailability  of  dynamic 
retention  rate  estimates.  In  short,  we  used  OS  data  to  substitute  for 
some  of  the  information  we  had  hoped  to  get  from  other  sources  (e.g. ,  the 
Oocxpational  Research  Data  Base,  or  ORDB) . 

In  any  event,  OS  data  served  many  purposes  in  the  prototype  RJS 
model.  Much  of  tlie  transition  matrix  for  ttje  current  pattern  for  each 
^lecialty  was  calculated  from  OS  data,  including  estimates  about 
job-to-job  flows,  reeiilistment/leave  rates,  rates  of  cross-training  into 
the  specialty  and  other  inodes  of  entering  the  speciedty.  OS  data  even 
sippleinented  the  J&THS  data  cxi  job-training  flews  when  there  were 
ineuiequate  returns  for  seme  smaller  job  types. 


2.1.8  Job-TIM  Associations 

OS  data  were  also  xjsed  to  determine  Job-TIM  associations.  TDS  job 
descriptions  must  be  written  in  terms  of  the  TIMs  produced  by  the  TCS, 
rather  than  in  terms  of  the  original  task  list  usually  used  for  such 
descriptions . 

We  used  the  CDDAP  system  to  generate  these  descriptions.  The  process 
begins  with  creation  of  a  Module  Title  File  from  a  standard  Task  Title 
File,  followed  by  creation  of  a  Modiile  Factor  File.  The  Module  Factor 
File  is  used  as  input  to  the  CDDAP  program  MOIDSET;  the  output  from  MODSET 
is  printed  using  the  PRIMOD  program.  Although  this  process  allows  one  to 
select  or  create  a  number  of  indices  to  represent  a  Job-TIM  association, 
we  selected  four  for  use  in  the  prototype  FUS.  All  are  straightforward 
aggregations  over  the  tasks  in  a  TIM  of  some  indices  familiar  to  anyone 
experienced  with  job-task  association.  Ihe  indices  appear  adequate  to 
describe  the  association  for  a  number  of  purposes,  and  include  the 
following: 

1.  Ihc  cum  across  people  cuxi  tasks  of  the  percent  time  spent 
performing  the  tasks  in  a  TIM. 

2.  A  running  cumulation  of  this  sum  percent  time  spent  index. 

3.  The  average  percent  time  spent  per  task  on  the  TIM  (i.e.,  the 
cumulation  /  #  people) . 

4.  The  average  percent  members  performing  across  tasks  of  the  TIM. 

Based  on  a  need  of  the  Training  Systems  Division  of  AFHRL,  a  facility  to 
compute  the  cumulative  sum  of  percent  time  spent  was  added  to  CDDAP. 

2.1.9  SIAM  and  New  Simulation  Software 

Another  concern  in  developing  dynamic  U&T  patterns  involved 
determining  vhat  kind  of  model  should  be  used  for  projecting  future 
requirements  and  consequences  for  current  and  alternative  U&T  patterns. 
Originally,  we  thought  it  likely  that  a  simple  single-step  Markov  model 
might  be  the  best  sort  to  use.  Eventually,  however,  we  becaine  interested 
in  using  techniques  of  dynamic  simulation  modeling. 

At  first,  we  explored  the  possibility  of  using  the  well-known 
simulation  language  SIAM  (Pritsker,  1986).  The  results  for  initial, 
simple,  imaginciry  U&T  patterns  were  encouraging.  The  language  was 
relatively  easy  to  use  and  seemed  adequate  for  the  job's  requirements. 

Unfortunately,  as  actual  U&T  data  were  accrumulated,  SIAM  seemed  less 
and  less  appropriate  for  this  particular  application.  SIAM  is  actually 
intended  to  simulate  the  experiences  of  discrete  entities  moving  through  a 
system  one  at  a  time,  with  little  or  no  iteration  through  the  same  or  very 
i>imilar  experiences.  FUS  job  flows,  however,  are  characterized  by  large 
numbers  of  entities  repeating  the  same  or  similar  experiences  many  times. 
For  present  purposes,  several  features  inherent  in  SIAM  became 
increasingly  cumbersome  liinitations  as  the  patterns  we  tried  to  model 
became  more  realistic  and  complex. 

We  adopted  the  solution  of  writing  new  simulation  software.  It  is 


actucilly  part  of  the  lOS,  but  the  new  software  was  written  to  be 
RJS-specific.  It  serves  the  functions  required  by  the  RJS  with  no 
additional  features  or  limitaticxis. 


2.2  The  Alternative  Utilization  and  Trainincf  Pattern 


lent 


2.2.1  Assetttolina  Alternative  Patterns 

Alternative  U&T  patterns  could  involve  changes,  relative  to  the 
current  pattern,  in  jobs  (e.g. ,  jcbs  could  be  ccnprised  of  different  sets 
of  TIMs,  or  could  be  performed  by  more  or  fewer  specieilty  merribers) , 
changes  in  the  number  or  types  of  training  states,  or  chcinges  in  the 
transition  probabilities  between  jobs  or  training. 


Vfe  began  work  on  eissembling  edtemative  patterns  in  concert  with  the 
effort  to  get  SMEs  to  define  the  current  pattern  (see  Section  2.1.6) . 

Many  of  the  SMEs  vho  helped  in  that  effort  were  ailso  asked  about  possible 
changes  in  the  structure  of  the  career  field.  Each  time  we  ^x>ke  with 
pecple  in  the  field  about  such  issues,  we  ed.so  solicited  informaticn  about 
Cctreer  field  problems  and  possible  solutions.  lAifortunately,  it  seemed 
difficult  for  many  of  the  people  we  were  working  with  to  understand  the 
sort  of  suggested  alternatives  we  were  after,  even  thou^  they  had  just 
been  considering  the  current  pattern.  In  additi(^,  suggested  alternative 
patterns  were  more  difficult  to  extract  from  811XX  and  423X1  incumbents 
than  frcm  the  328X4  and  305X4  members,  apparently  because  many  in  these 
career  fields  were  very  satisfied  with  the  overall  current  ccaifiguration. 
It  is  often  frustration  with  identifiable,  fixable  problems  that  spawns  a 
number  of  alternative  suggesticais. 

Most  of  the  alternatives  represented  in  the  prototype  system  were 
suggested  by  various  sources  during  visits  to  operational  bases,  to  the 
Office  of  Security  Police  at  Kirtland  AFB,  and  throu^  a  ^)ecial  session 
during  the  811XX  U&T  Workshop  at  Lackland  AFB  in  May  1986.  we  reduced 
ideais  to  descriptive  naunrative  and  acccnparying  flow  dieigram,  euid 
described  the  current  U&T  pattern  in  the  same  way. 

2.2.2  Products  of  the  Alternative  U&T  Pattern  Oopponent 

One  product  of  this  ccnponent  is  the  set  of  alternatives  itself.  In 
many  cases,  the  edtematives  represent  interesting  restructurings  of  work 
and  training  in  a  career  field. 

Another  product  is  the  output  of  a  simulation  run,  much  like  that 
provided  for  each  current  U&T  pattern  but  now  performed  for  each 
ciltemative  in  each  specieilty.  It  would  be  desirable  to  have  these  data 
available  during  the  management  preference  surveying  process;  this  was  not 
possible  for  the  prototype  effort.  Ihe  projections  of  consequences  mi^t 
become  an  important  factor  in  some  managers'  preferences.  Simulation  runs 
can  show  that  sente  alternatives  are  simply  not  viable  without  Considerable 
additional  changes.  For  example,  requiring  that  certain  jobs  not  be 
performed  by  first-terra  individueds  can  leave  them  badly  undermanned 
unless  current  retenticmi  rates  can  be  improved  dramatically.  Such 
informaticn  could  be  important  to  anyone  who  ocnslders  this  pattern  a 
potential  solution  to  another  existing  problem. 
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2.3,1  Survey  Administration 


The  U&T  pattern  narratives  and  diagrams  were  assembled  into  booklets, 
one  for  eadi  specialty,  and  administered  face-to-face  (with  a  handful  of 
exceptions)  to  functional,  personnel,  trainiryj,  and  other  kinds  of 
managers  and  senior  personnel  in  each  specialty  during  July  1987.  The 
aralysis  and  results  of  this  surveying  effort  were  provided  to  AFHRL  as  a 
Data  eincVor  Analysis  Sumtiary.  Briefly,  the  effort  succeeded  in  achieving 
its  objectives.  Moreover,  interrater  agreement  ranged  frcan  acceptable  to 
very  hi<^  for  each  specialty.  Still,  several  suggestions  were  offered  to 
improve  the  process.  Ihese  included  the  follcwing: 


1.  Perform  a  pilot  study  for  each  survey.  This  is  mainly  intended  to 
improve  the  final  survey.  Several  respondents  found  it  easy  to 
suggest  new  patterns  by  altering  or  reacting  to  some  otheirs  already 
presented  to  them. 

2.  Be  more  explicit  in  requesting  comments  about  the  survey  or 
eliciting  additional  alternative  patterns. 

3.  Administer  this  survey  in  a  face-to  face  interview  session  whenever 
possible. 


4.  Include  a  wide  veiriety  of  managers  in  the  sample.  During  this 
prototyping  effort,  manpower,  personnel,  and  training  managers  were 
consistently  underrepresented. 

5.  Performing  additional  eiploratory  statistical  analyses  might  be 
valuable,  for  exanple,  to  identify  lander lying  conponents  inportant 
to  acoeptcince  of  a  pattern. 


2.3.2  Products  of  the  Metnaaement  Preferences  Component 

Ihe  main  product  of  this  ccnponent  is  Manager  Preferences.  These  are 
ejpressed  in  the  form  of  ratings  on  a  l-to-9  scale  (1  showing  low 
preference,  9  shewing  hi<^  preference)  in  the  prototype.  The  preferences 
may  be  aggregated  at  different  levels,  as  necessary.  The  prototype  TDS 
presents,  for  each  alternative,  overall  average  ratings  for  the  entire 
sample  of  managers,  and  also  numbers  of  respondents  and  average 
preferences  for  each  "type"  of  respondent. 


2.4  Evaluation  of  the  FUS 

We  also  evaluated  the  FUS  develcped  for  the  prototype.  This  involved 
another  survey  effort,  which  was  cetrried  out  simultaneously  with  the 
Management  Preferences  Survey;  the  results  were  also  reported  in  the  Data 
and/or  Analysis  Summary. 

The  Evaluation  Survey  consisted  of  five  questions  and  opportunities 
for  additional  comments.  Each  question  was  answered  by  cheeddng  either 
"Yes,"  "No,"  or  "Uncertain."  Question  1  asked  about  the  usefulness  of  the 
method  by  which  the  survey  presented  the  current  U&T  pattern  description; 
Question  2  aisked  if  the  description  was  accurate  per  se;  Question  3  asked 
if  the  description  accounted  for  personnel  flew  throui^out  the  entire 
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career;  Question  4  asked  if  the  alternative  patterns  v^ere  understandable; 
and  Question  5  asked  if  there  was  a  better  way  (than  that  instrument)  to 
gather  preference  data.  People  answering  "Yes"  to  Question  5  were 
specifically  asked  for  their  ideas  in  that  regard,  but  re^xndents  were 
encouraged  to  provide  additional  ccmnents  on  any  or  all  of  their  answers. 
As  with  preferences  data,  these  respc^ises  were  ocnpiled  and  presented  in 
two  tables:  one  for  overall  tallies,  and  cxie  vAiich  tabled  re^xxises  by 
type  of  manager. 

Evciluation  results  were  generally  very  favorable.  Most  found  U&T 
patterns  and  alternatives  very  useful  and  understandable.  Ifiifortunately, 
there  were  very  few  cements  to  e^q^lain  any  negative  reactions  there 
were.  Ihe  main  reoomnendaticxi  offered  about  this  survey  was  that 
respcMTdents  be  asked  more  directly  emd  ^ecificzilly  to  e}g)lain  viiat  they 
dislike  about  the  effort  or  the  survey  itself. 


Overed-l,  the  FUSHxiilding  effort  has  been  successful.  Ihe  corpleted 
model  2^:pears  ocherent,  internally  ocaisistent,  and  reac^  for  extended 
testing.  Saisltlvlty  analyses  have  2dj:eac^  been  planned  vMch  should 
unccrver  and  oomect  any  hidd^  flaws  the  model  or  ijiplentented  system  may 
have. 

The  prototype  system  will  be  ocnfigured  to  produce  the  reports  and 
other  information  vMch  IDS  staff  meaxbers  from  AFHRL,  McDonnell  Douglas, 
and  GCMSAD  Research  Corporation  have  agreed  are  likely  to  be  most  useful 
in  meeting  FDS  requirements.  Many  of  these  were  reviewed  by  the  TD6 
Advisory  Panel  at  the  January  1987  progress  review.  It  is  likely  that 
edditional  useful  products  can  be  assenhled  from  existing  data,  if  user 
ocianunities  require  further  reports.  Additional  data  can  be  collected  as 
warranted. 

On  the  other  hand,  one  can  wish  for  something  else,  perhaps  something 
better.  In  collecting  some  of  the  data  emd  developing  some  of  the  methods 
we  have  tried  various  alternatives  and  ultimately  used  the  best  that  were 
at  hand.  But  there  are  other  methods  that  were  not  tried,  for  various 
reasons.  Some  methods  were  considered  hi^  risk;  they  would  likely 
produce  better  data  if  they  worked,  but  would  take  a  lot  of  time  and  money 
without  particularly  hi^  assurance  that  they  would  work.  In  other  cases, 
pressing  demands  did  not  eLLlcw  the  luxury  of  additional  effort  vhen  a 
workable  method  was  at  hand.  In  still  other  cases,  we  had  no  control  over 
events  or  access  to  source  that  mi^t  have  yielded  better  data. 

Ihe  purpose  of  this  section  is  to  provide  observations  cn  the  quality 
of  the  present  model  or  data,  suggest  specific  improvements  vhen  possible, 
and  discuss  pertinent  issues  vhen  ^»cific  suggestions  can  or  should  not 
be  made.  In  any  event,  many  possible  changes  to  the  PUS  process  would  be 
more  a  matter  of  policy  than  of  need. 


3.1  Ihe  current  U&T  Pattern  Ocnponent 

Ihe  greatest  number  of  problems  and  suggestioais  concern  this 
ccBponent.  Ihis  is  not  surprising;  the  current  U&T  pattern  is  the  key  to 
the  entire  subsystem.  Also,  mudh  of  the  PCS  requires  accurate  inpit  from 
the  FUS  if  it  is  to  work  properly. 

3.1.1.  jgi?^ryping 

Ihe  present  method  of  job-typing  basically  consists  of  reviewing  and 
revising  the  OSR  job  types  as  necessary.  Ihis  approach  was  judged 
tentatively  acceptable  in  adl  four  prototype  TD6  ^jecialties,  but  there 
are  some  shcnrtcomings.  Whether  improvements  on  the  present  methods  aune 
required  depends  on  how  well  the  present  methods  serve,  and  vbether  there 
are  any  future  extensions  to  the  present  FUS. 
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3. 1.1.1  Ncai-Hierarchical  Clvtsterincr.  Not  eill  people  are 
assigned  to  specific  jobs  under  the  present  hierarchical  clustering 
methodology.  Ihese  people  were  placed  in  a  "no  job  type”  category  for  the 
prototype  effort.  Hiis  approach  may  present  no  particular  problem,  and 
could  be  used  for  some  time  to  come.  Ihe  "no  type"  category  can  be 
described  in  terms  of  TIMs,  linked  to  training  requirements,  ascribed  a 
retenticxi  rate,  etc.  like  any  actual  job  type.  Ihe  data  fed  to  the  ECS  as 
a  result  are  peirfectly  good.  But  there  are  questions  about  how  aocurately 
hierarchical  clustering  produces  representative  job  types,  and  the  typicail 
failure  of  the  procedure  to  assign  vp  to  20%  of  an  AES's  members  is 
syrnptomatic  of  possible  troubles.  The  matter  deserves  attention  if  the 
RJS  is  to  be  interpreted  as  a  hi^Hy  accurate  portrayal  of  the  current 
system. 


Research  and  testing  of  a  new  norHtuerarchiced  epproach  is  underway  at 
APHRL.  Indeed,  IDS  requirements  have  provided  seme  of  the  inpetus  for 
these  investigations  and  perhaps  hastened  their  eventued.  success.  The 
potenticil  advantage  comes  from  the  fact  that  these  non-hierarchical 
clusters  can  form  and  reform  more  easily.  With  hierarchical  methods,  an 
entity  is  captured  for  good  once  it  is  pulled  into  a  group.  The 
formatlcxi,  at  a  later  stage,  of  a  possibly  more  similar  group  has  no 
effect  CHI  that  entity.  The  entity,  in  turn,  alters  the  overall  character 
of  the  groip  vhich  captures  it,  affecting  vhich  other  entities  are  later 
captured  by  that  groip.  This  situation  is  relaxed  under  non-hierarchical 
clustering.  As  a  result,  one  recent  stuc^  (Mitchell  &  Ihedoi,  1985)  found 
that  the  nuiitoer  of  unclustered  people  dropped  dramatically,  relative  to 
the  hierarchical  approach. 


The  impact  of  this  issue  on  the  HJES  is  not  clear,  but  several 
improvements  mic^t  result  from  using  a  different  clustering  scheme.  For 
example,  the  issue  of  first-term  versus  career  job  types  could  be 
re-investigated.  The  problem  involved  in  splitting  the  sample  was  that 
the  effects  were  not  always  easily  interpretable.  In  seme  instances, 
members  of  a  vhole-sample  job  type  wound  up  a  considerable  distance  from 
one  another  on,  say,  the  first-term-only  dieigram.  They  mi^t  not  ocme 
together  again  until  joining  in  a  big  grxxp  that  had  many  other  members 
and  intra-group  similarity  indices  too  low  to  indicate  that  the  group  was 
a  job  type  according  to  the  usual  guidelines.  What  is  onu  to  make  of 
this,  especied-ly  vhen  further  examination  reveed.s  that  the  subgroups  are 
still  not  very  different  in  job  descriptions?  They  mi^t  overlap  greatly 
in  tasks  performed,  but  spend  very  different  amounts  of  time  on  subsets  of 
those  tasks.  Moreover,  in  some  instances  one  could  also  identify  very 
ocnparable  patterns  between  career  and  first-term  diagrams,  with  an  almost 
ccnplete  mapping  of  groups  and  subgroups. 


One  possibility  is  that  hierarchical  clustering  may  overelaborate  the 
heterogeneity  of  some  career  fields  for  the  ixmxises  of  the  TT36.  This  was 
very  possibly  the  case  for  the  FUS  model  of  AES  328X4,  in  the  opinion  of 
the  analyst  vho  performed  these  studies.  Even  seme  of  the  job  types 
eventually  used  were  not  terribly  different  frem  one  another,  except  in 
the  amounts  of  time  spent  performing  the  same  tasks.  One  could  make  a 
good  case  that  such  jobs  are  not  differmit  in  terms  of  training  required. 
The  cxxtplication  is  that  because  "jobs"  are  oenprised  of  ta^  clusters, 
different  jobs  in  the  current  pattern  may  not  require  different  training 
in  seme  cases;  but  in  cither  cases,  wi^dung  the  same  job  on  different 
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aircraft  mi^t.  Also,  although  unclustered  people  are  considered  by  the 
TDS,  in  practice  their  representation  is  limited  since  they  are  obscured 
by  membership  in  a  very  large  ("no  job  type")  group.  When  cutoffs  are 
established  that  limit  the  number  of  Tn»Is  us^  to  describe  a  job,  the 
final  result  is  that  many  people  may  not  be  represented  at  all. 

3. 1.1. 2  Additional  Classification  Approaches.  There  cire  other 
possibilities  for  changing,  and  possibly  inproving  (for  TDS  purposes),  the 
job  types.  A  method  could  be  devised  vhereby  individuals  are  clustered 
according  to  the  degree  to  vhich  they  perform  entire  TIMs,  rather  than 
tcisks.  In  a  case  such  cis  423X1,  in  vhich  there  vras  a  single  job  type  that 
accounted  for  over  80%  of  the  specialty,  it  would  be  worthvhile  to  attempt 
a  discriminant  analysis  to  classify  people  according  to  additional 
background  variables.  This  might  allow  separation  of  the  large  job  into 
reasonable-sized  jobs  according  to  the  aircraft  worked  on,  for  exanple. 
Such  jobs  mi^t  not  meet  CMC's  needs,  but  the  usefulness  of  such  job  types 
for  purposes  of  the  TDS  remains  an  cpen  question. 

3. 1.1. 3  SME  Review.  In  any  case,  the  ideal  situation  for  final 
jcto  selections  in  an  AFS  would  allow  for  review  and  comment  by  several 
SMEs  vho  have  been  trained  to  the  point  where  they  understand  and  endorse 
the  FUS  effort,  at  least  to  some  degree.  Ihe  prototype  job  types  were 
reviewed  by  SMEs,  but  they  were  seldom  knowledgeable  about  the  TDS  or  the 
job-typing  process. 

3. 1.1. 4  Changes  in  Specialties.  AFS  305X4,  one  of  the 
specialties  considered  in  the  prototype  TDS  effort,  has  been  preparing  for 
substantial  changes  over  the  last  couple  of  years.  Several  old  shreds 
were  about  to  be  replaced  by  new  ones,  with  corresponding  major  changes  in 
equipment  cind  training  courses.  This  presented  certain  problems 
throu^out  the  current  effort,  and  one  result  is  that  the  present  TDS 
305X4  configuration  represents  the  old  form  of  the  specialty.  The  issue 
of  hew  to  deal  with  similcir  situations  in  the  future  and  the  related 
issue  of  whether  it  is  desirable  to  begin  a  TDS  study  on  a  specialty  that 
may  soon  undergo  substantial  change  both  deserve  further  research.  It 
may  be  that  the  TDS  can  be  particularly  helpful  if  additional  planning  is 
done  cind  the  study  is  conducted  with  additional  or  somewhat  different 
goals  in  mind. 

On  the  other  hand,  we  have  been  informed  by  personnel  at  Offutt  AFB  of 
planned  changes  to  AFS  328X4 ,  at  least  in  Strategic  Air  C3ommand  (SAC) . 

The  prc^x3sed  changes  generally  involve  combining  duties  with  those  of 
other  specialties.  The  TDS  can  be  useful  in  this  situation,  but  its 
usefulness  may  be  limited  to  examining  the  impact  vpon  the  328X4  part  of 
the  new,  combined  specialty.  This  is  because  (with  a  few  exceptions)  the 
system  is  presently  designed  to  consider  changes  only  in  a  single 
specialty  in  isolation.  However,  the  TDS  could  be  expanded  to  consider 
multiple  AFSs  simultaneously.  Any  such  extensions  should  proceed  with  a 
view  teward  modeling  the  sort  of  changes  now  plemned  for  AFS  328X4. 

3. 1.1.5  Coordinating  0S-TD6  Efforts.  Many  advantages  could 
result  fresn  close  cooperation  between  the  TDS  process  eind  the  OS  process. 
Some  of  the  job-typing  problems  discussed  above  mi^t  be  resolved  if  more 
or  less  specific  task  lists  were  used  in  the  OS  (for  a  discussion  of  this 
issue,  see  Mitchell  et  al.,  1986).  This  is  not  to  suggest  that  TDS 
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cxsnsideratic»is  should  drive  the  06  process,  but  ideally  a  hi^  level  of 
mutual  decision-making  would  exist  between  the  two  efforts.  Also,  06  and 
TD6  studies  should  probably  be  begun  simultaneously  to  maximize  this 
coordination,  and  a  program  should  be  worked  out  for  coordinating  and 
intertwining  06-TD6  data  gathering  and  analysis  steps.  It  mi^t  also  be 
beneficial  to  assign  a  single  analyst  to  do  the  entire  sequence  of  TD6 
work  on  any  given  speciality.  Uiis  would  adlow  analysts  to  develop 
proficiency  in  all  aspects  of  the  TD6  and  to  work  closely  with  the 
counterpart  analyst  working  on  the  06  effort,  and  it  would  minimize 
coordination  problems  with  TD6  users,  CMC,  and  other  entities.  It  is 
possible  that  the  RCS  mi^t  require  a  second  an2d.yst  with  ocxisiderable 
expertise  in  economics,  but  the  minimum  fectsible  nunber  should  probably  be 
used  in  any  event. 

3.1.2  U&T  Flow  Patterns 

It  is  also  possible  that  the  methods  described  in  this  paper  will  be 
judged  suitable  for  the  operational  TD6.  As  with  job  types,  any  nunber  of 
alternative  methods  are  possible.  Policy  decisions  and  considerable  TD5 
testing  will  establish  v^ther  tl'^se  cure  or  cure  not  needed. 

3. 1.2.1  6inDlifvincf  Assunctions  in  the  Prototype.  Ihe 
transition  matrix  for  each  specicdt^  iiiplicitly  reflects  a  nunber  of 
simpli^ing  assunpticxis.  One  exanple  is  the  lack  of  job-to-job 
transiticais.  Instead,  an  abnegation  node  was  used  in  each  matrix  and 
corresponding  simulation.  Job-changers  are  sirply  aggregated  and 
reassigned  to  new  jobs  randcmly  according  to  TAFM5  cohort.  Actually,  this 
approach  closely  models  the  true  assignments  process  as  it  has  been 
described  to  us  r^)eatedly  by  SMEs.  Hcwever,  if  job-to-job  transitions 
are  desired,  they  can  probably  be  calculated. 

The  J&IHS  effort  provided  sane  vital  information  that  could  not  be 
obtained  elsewhere,  but  for  various  reasons  (e.g. ,  small  or  zero  survey 
returns  for  many  jobs,  lack  of  time  for  analysis  anx  the  surveys  finedly 
began  to  return  fTcm  the  field)  was  vmable  to  sipport  some  possible  RJ8 
features  such  as  job-to-job  transitions. 

Close  cooperation  between  the  06  ^uld  TD6  processes  would  serve  to 
alleviate  many  of  these  problems.  Also,  the  J&IHS  effort  should  begin  a 
TI16  study,  rather  than  coming  toward  the  end  of  one.  With  sufficient 
cooperation,  virtually  the  entire  06  sample  would  edso  have  the 
opportunity  of  supplying  informatioi  required  for  the  HB.  In  addition, 
the  larger  sanple  would  probably  edlow  a  certain  degree  of  "cleaning**  of 
the  data.  For  exanple,  in  the  prototyping  effort  the  J&IHS  sample  was  so 
stncill  that  we  needed  to  use  all  the  job  histories  available.  In  a  few 
cases,  this  meant  that  we  were  using  subjective  job  histories  supplied  by 
people  vho  had  failed  to  pick  their  present  06  job  correctly,  or 
indicated  that  they  changed  jobs  every  couple  of  months  on  tte  average. 

In  some  sense,  these  people  have  shown  either  that  their  judgment  or 
adherence  to  survey  instructiois  is  questionable,  or  that  they  really  are 
atypical  for  sane  unknown  reason.  With  a  much  larger  sanple,  we  mic^t  be 
able  to  afford  the  luxury  of  establishing  criteria  that  determine  vMch 
responses  should  be  kept.  Perhaps  oily  those  vho  correctly  identify  their 
present  job  (according  to  the  06R)  would  be  included,  or  only  those  who 
both  pick  the  correct  job  and  do  not  indicate  an  average  of  more  than  one 
jcb  transition  per  year.  Just  vhat  are  or  are  not  good  criteria  deserves 
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investigation;  without  sound  justification  there  is  danger  of  keeping  cnly 
those  responses  that  fit  one's  eog^ectations. 

3. 1.2. 2  SHE  Input.  Anothear  matter  of  potential  iiEportanoe  to 
the  J&IfS5  involves  early  input  frtxn  knowledgeable  SMEs.  Ihe  prototype 
IDS,  for  exanple,  contains  no  information  about  IlATCCM-^)ecific  training 
courses.  We  were  sirply  laiable  to  obtain  sufficient  information  about 
such  courses.  Even  now  their  inportanoe  for  the  US  is  unresolved;  it  may 
be  that  they  are  best  omitted.  As  another  exanple,  the  prototype  does  not 
include  information  regarding  "generic"  FIDs,  vhioh  receive  students  from 
any  number  of  AFSs.  Examples  indiade  FIDs  that  teach  about  digital 

high-rellEQsility  soldering,  serving  as  an  OIT  supervisor,  and 
the  like.  On:  present  impression  is  that  they  should  be  included  because 
they  provide  a  lot  of  apparently  important  training.  It  was  too  late  to 
determine  vhich  TlMs  are  taught  in  such  courses  fcy  the  timie  surveys  were 
returned  and  we  realized  their  importance. 

In  some  ways,  this  situation  reflects  another  important  instance  of 
oversimplification.  The  prototype  HJS  implicitly  assumes  that  any 
training  not  provided  in  one  of  the  training  states  included  in  the  model 
is  provided  by  OJT.  Thus,  it  is  important  to  determine  the  optimum  number 
and  type  of  courses  to  include,  so  as  not  to  badly  overestimate  the  CXTT 
burden.  The  epproach  used  in  the  prdtdtypR  tends  toward  malcing  reality 
fit  a  model.  Probably  the  best  way  to  resolve  this  problem  is  to 
interview  a  wide  variety  of  working  AFS  members  as  part  of  the  J&IHS 
assembly  process.  Incumbents  are  probably  the  best  jixiges  of  where  they 
reoelve  the  training  important  to  their  current  work  and  future  career 
a^irations,  and  should  be  closely  involved  with  the  entire  process  of 
planning  the  survey  effort  and  constructing  the  survey  itself. 

3. 1.2. 3  Streamline  Survey  Processes.  The  mechanics  of  the  Job 
and  Training  History  Surveys  oould  use  additional  investigation  and 
pilot-'testing.  Ideally,  initial  construction  would  assemble  the 
job/tralning  state  lisite.  This  oould  be  followed  by  pilot-testing  to  see, 
for  example,  if  a  decision-tree  stmcturing  mlc^  aid  respondents  in 
picking  jobs  for  each  time  period.  Also,  we  might  obtain  better  results 
if  people  were  askad  to  relate  training  oourses  to  particular  jobs 
directly  in  their  own  histories.  The  present  format  required  that 
respondents  list  jobs  and  training  sp>arately,  and  the  two  were  linked 
eifterward  by  assuming  that  training  given  in  ary  period  was  pertinent  to 
the  job  occupied  in  that  period.  This  may  or  may  not  be  valid.  In  ary 
event,  the  case  has  alreanj^  been  made  for  asking  the  people  what  th^ 
thlilk  was  their  actual,  pertinent  training  for  each  job. 

3. 1.2. 4  Future  ExtensiCTis.  An  FUS  model  oould  potentially 
beorme  very  exact  and  complex,  giv^  sufficient  information  from  a  leirge 
sample.  To  give  a  single  example,  one  mic^  be  able  to  model  job  flow 
among  various  oomwands  and  bases.  A  truly  dynamic  model  mi^t  show 
fluctuations  in  flow  across  bases  or  other  entities  aooording  to  the 
simulation  start  time  selected.  Or,  the  model  might  devise  and  use 
abstractions  such  as  "represertatlve  bases"  to  examine  differences,  ihe 
main  purpose  of  such  a  ocmplex  model  would  be  to  help  the  PCS  establish 
the  meet  accurate  oost/resouroe  requirements  estimates  possible. 


Another  important  extension  of  the  FUS  would  involve  fiurther  use  and 


coordination  of  different  sources  of  data.  For  example,  the  prototyping 
effort  involved  only  limited  use  of  the  QRDB,  euxJ  left  unresolved  many 
issues  concerning  coordination  of  that  data  base  with  additional 
information  from  the  OS  TDS-^)ecific  surveying  efforts,  and  other 
personnel  and  training  data  beises. 

3.2  The  Alternative  U&T  Pattern  Component 

Little  needs  to  be  said  about  this  component.  The  effort  went 
smoothly  and  the  information  gathered  appears  satisfactory,  unless  it 
beccmes  desirable  in  the  future  to  alter  the  nature  of  the  component 
considerably. 

Moreover,  some  pertinent  ocmments  have  been  made  earlier  in  this 
paper.  We  have  previously  discussed  the  observation  that  SMEs  afpecur  most 
able  to  offer  alternative  patterns  if  they  have  examples  of  eiltemative 
U&Ts  in  front  of  them.  These  eiltematives  need  not  be  sophisticated; 
crude  examples  vtiich,  say,  simply  eliminate  a  training  setting  mi^t  serve 
the  purppse.  This  and  a  few  other  strai^tforward  changes  to  the  ways  in 
vhich  alternate  patterns  eure  designed  could  be  very  helpful. 

Another  suggestion  can  be  offered.  It  md^t  be  advisable  to  edlcw 
J&IHS  respondents  to  suggest  adtematives.  The  possibility  of  ausking  them 
about  very  jc±)-specific  training  has  been  discussed.  Additicxal  questions 
could  ask  about  additional  desirable  training  for  each  jcA>  that  was  not 
received,  or  about  unnecessary  training  that  was  received.  Respondents 
could  also  express  cpinions  on  combining  or  dividing  jobs.  Analysis  of 
the  resulting  data  could  suggest  alternatives  that  attempt  to  resolve 
prc±»lene  perceived  by  lower-echelon  personnel  in  the  field. 


3.3  The  Management  Preferences  Component 

Again,  some  potenticilly  important  improvements  to  this  component  have 
already  been  discussed.  On  the  other  hand,  implementing  some  of  these 
suggestions  in  conjunction  with  each  other  could  enhance  the  value  of  this 
component  considerably. 

3.3.1  Simulation  Runs 

Simply  having  the  results  of  current  and  alternative  U&T  pattern 
simulation  runs,  as  well  as  various  PCS  and  lOS  products,  available  for 
SME  inspection  would  allow  them  to  make  much  more  inform^  preference 
decisions. 

3.3.2  Statistical  Modeling 

We  have  alreacty  mentioned  the  possibilil^  of  using  statistical 
modeling  techniques  (for  example,  multidimensional  scalii>g)  to  explore  the 
components  of  managers'  preferences. 

3.3.3  SME  Comments 

The  report  on  the  prototype  effort  in  this  subsystem  contains  little 
information  about  the  comments  made  by  respondents,  primarily  because  few 
ccamments  were  -offered.  It  would  probably  be  best  to  ask  directly  vhat 
people  like  or  dislike  about  einy  given  pattern,  more  or  less  forcing  em 
answer.  At  present,  a  user  will  know  the  results  of  the 


prefesrenoe-gathering  effort;  however,  e}^)erienoed  builders  of  decision 
support  systaoBS  reedize  that  this  sort  of  information  is  often  disregarded 
if  presented  in  isolaticsi.  The  user  should  be  provided  an  eidequate  frame 
of  reference  to  increase  acceptability. 

In  short,  we  mi^t  determine  vAuch  patterns  are  preferred,  find  out 
vAiy  they  eune  preferred,  and  find  out  what  problems  their  implementation 
oculd  bring.  If  the  process  could  be  iaplanented  successfully,  iterative 
revisions  and  preference  surveyings  could  create  adtemative  patterns  that 
various  managers  would  like,  that  would  be  practical,  and  that  would  be 
cheaper  to  maintain.  Iliis  information  could  enhance  the  FUS  contribution 
to  the  Air  Force  and  Air  Force  training. 
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